This invention relates to the field of computer systems. More particularly, a system and methods are provided for accurately modeling and emulating input/output performance of an application in different computing environments or with different operating parameters.
Various methods are traditionally employed to generate benchmarks for characterizing an application's likely performance in a particular environment. For example, an application's input/output (I/O) may be traced or modeled to predict the application's performance under different operating conditions, to determine capacity requirements, to determine how to increase its performance, and/or for other reasons.
Through tracing, an application's I/O activity is recorded to learn how it operates in a given computing environment (e.g., a particular file system operating with a given set of parameters). That trace, which may be very large, may reveal extensive system and operational characteristics of the application, but is generally tied to the specific system configuration on which the application was traced. The trace may allow accurate benchmarking of the application with that system configuration, but will produce inaccurate results if applied to different configurations or under different operating conditions or parameters (e.g., different file systems, different I/O parameters). In addition, the trace cannot capture I/O dependencies between application threads or processes.
A model-based method of testing or evaluating application performance involves the creation of a model representing the application's mix of input/output operations (e.g., 50% reads, 40% writes, 5% opens, 5% deletes). After the model is created, however, it is used to merely simulate the application's performance in a different environment. Instead of generating and evaluating actual input/output operations, those operations are simulated to produce statistical estimations of the application's performance. Thus, the resource consumption and latency of the application's I/O can only be estimated, and predictions of the application's performance in other environments will be suspect.
In summary, depending on the manner in which an application is characterized, the resulting predicted performance may or may not accurately represent the application's actual I/O behavior, and may or may not be flexible enough to reflect the application's performance in different computing environments.
For example, traditional methods of benchmarking cannot accurately represent a database application that has a strong dependence on a single thread or process (e.g., its transaction log writer). Tracing or simulated modeling might capture the different types of I/O operations, but cannot capture the effect that additional latency on that single thread will have on other threads.
Also, depending on the complexity of the application being benchmarked, measuring its input/output performance may require substantial expertise. For example, testing a file system for a DBMS (Database Management System) may require sufficient database administration skills to not only configure the computing environment for the DBMS, but also to apply a set of on-line transaction processing benchmarks, such as TPC-C (offered by Transaction Processing Performance Council).
A significant amount of setup time may thus be required to enable a single performance measurement. And, during testing, computer system resources must generally be dedicated to operation of the application and collection of performance data. To test different computing environment configurations or operating parameters requires additional time for reconfiguration.
Thus, a system and a method are needed for benchmarking an application's I/O performance so as to allow the application's performance under different operating environments or conditions to be accurately tested.